%
% $Id: appD.tex,v 1.20 1996/05/08 00:04:59 radford Exp $
%
\newpage

\section{GLOSSARY OF DELVE TERMINOLOGY}\label{app-glossary}
\thispagestyle{plain}
\setcounter{figure}{0}
\chead[\fancyplain{}{\thesection.\ GLOSSARY OF DELVE TERMINOLOGY}]
      {\fancyplain{}{\thesection.\ GLOSSARY OF DELVE TERMINOLOGY}}

\vspace{4pt}

\small
\begin{list}{}{%
\setlength{\itemsep}{0in}%
\setlength{\leftmargin}{2.25in}%
\setlength{\labelsep}{0in}%
\setlength{\labelwidth}{2.25in}}

\item[\bf absolute-error loss \hfill]
A loss function for regression tasks in which the loss is the
absolute value of the difference between the guess and the
target.  When there is more than one target, the absolute loss 
is the sum of such absolute differences for all the targets.

\item[\bf angular attribute \hfill]
An attribute whose value is an angle or some other circular
quantity, such as time-of-day.  By default, such attributes
are encoded as the sine and the cosine of the equivalent angle,
so as to avoid introducing an artificial discontinuity.

\item[\bf artificial dataset/prototask \hfill]
A dataset generated by a program (usually with a random component)
on the basis of some mathematical specification, without any
connection with a real-world problem.  Prototasks based on 
such datasets are also referred to as artificial.

\item[\bf assessment dataset \hfill]
A dataset that is recommended for use in formally assessing
learning methods.

\item[\bf attribute \hfill]
One of the quantities associated with each case in a dataset.
The dataset specification classifies attributes as {\bf controlled} 
or {\bf uncontrolled}, according to how their values were determined.  
The prior information for a task will characterize attributes
as {\bf binary}, {\bf nominal}, {\bf ordinal}, {\bf integer}, {\bf angular},
or {\bf real}.

\item[\bf binary attribute \hfill] 
A categorical attribute that can take on exactly two possible values 
(not counting missing values) --- for example, an attribute with possible 
values of ``male'' and ``female'', or one with values of ``0'' and ``1''.

\item[\bf categorical attribute \hfill]
An attribute that takes on values from some finite set.  The targets for
a classification prototask must be categorical.  The prior information for 
a task further characterizes categorical attributes as {\bf binary},
{\bf nominal}, or {\bf ordinal}, and may designate one of the values
as {\bf passive}.

\item[\bf case \hfill]
A collection of attribute values that all apply to the same thing.  For
example, in a dataset of medical tests on patients, a case might consist
of all the test results for a particular patient.

\item[\bf censored value \hfill]
An indication of the value for an attribute in a case that says
only that the value is known to be greater than or equal to (or less
than or equal to) a specified value.  In DELVE dataset files, a
censored value is recorded as ``{\em number\/}:'' (if the actual value is
greater than or equal to {\em number\/}), or as ``:{\em number\/}'' (if the 
actual value is less than or equal to {\em number\/}).

\item[\bf classification prototask/task \hfill]
A prototask (or task) in which all the target attributes are
{\bf categorical}.

\item[\bf controlled attribute \hfill]
An attribute whose values were fixed by the investigators who gathered
the data.  For example, the amount of fertilizer applied to an agricultural
test plot would likely be a controlled attribute.

\item[\bf common testing scheme \hfill]
An experimental set-up in which a single common test set is used to
assess the performance of a method with all the training sets; distinguished
from a {\bf hierarchical} testing scheme.

\item[\bf commonality index \hfill] 
An integer that may be associated with a case, indicating that the
case has something in common with the other cases with the same
commonality index.  For example, in a dataset where a case records
features of a handwritten digit, all the digits written by one person 
might have the same commonality index.

\item[\bf cultivated dataset/prototask \hfill]
A dataset that comes from a real-world source, but has no real-world
context, having been collected or selected for the purpose of 
creating a DELVE dataset rather than from any genuine interest.  Natural
datasets that have been modified in some way, such as by adding extra
noise, are also in this class.  Prototasks based on cultivated datasets are 
also classified as cultivated, as are prototasks that are based on natural 
datasets but which have little resemblance to the original purpose for 
which the data was gathered.

\item[\bf dataset \hfill] 
A collection of data, consisting of a number of {\bf cases}, each
associated with the values of several {\bf attributes}.  Datasets are
classified as {\bf natural}, {\bf cultivated}, {\bf simulated}, or
{\bf artificial} according to the data's relationship to the real
world.  DELVE also distinguishes among {\bf development datasets},
{\bf assessment datasets}, and {\bf historical datasets}, on the basis
of recommended usage.

\item[\bf default encoding \hfill]
The encoding of an attribute that DELVE will use by default
if a particular learning method does not specify otherwise.
The default encoding is based on the {\bf prior information}
for the task.

\item[\bf dependency (between cases) \hfill]
A situation where knowledge of the values of the targets in one case
would be informative regarding the values of the targets in other
cases with the same {\bf commonality index}, or that are nearby in a
{\bf sequential prototask}.  Here, it is assumed that the inputs in all cases
are already known, and that the true nature of the general relationship 
between inputs and targets is also fully understood --- ie, the dependency 
is between the ``noise'' or ``residuals'' in the related cases (the
part of the variation not explainable by the relationship between inputs
and targets).

\item[\bf development dataset \hfill]
A dataset that is recommended for use in developing learning methods.
To avoid bias, such datasets should not also be used in formal assessments
of performance.

\item[\bf encoding (of an attribute) \hfill]
The way that DELVE represents the value of an attribute (usually as one or
more numbers) when generating data files for task instances.  The 
encoding to use is part of the specification
of a learning method, but DELVE provides a 
{\bf default encoding} that will often be appropriate.

\item[\bf estimated expected loss \hfill]
An estimate for the expected loss of a learning method on some task,
based on the results of a learning experiment.  At present, DELVE's
estimates are simply the average loss over training sets and test cases
tried.  Each estimate has an associated {\bf standard error}, that is
indicative of its likely accuracy.

\item[\bf expected loss \hfill]
The expected performance of a {\bf learning method} on some {\bf task} 
as judged by a specified {\bf loss function}, the expectation being with 
respect to random selection of a training set and a test case.  Put another 
way, the performance the method would achieve on average if it were
applied a great many times to training sets and test cases obtained from
the same source as the actual dataset.  Note that the true expected loss cannot 
be determined exactly, but an {\bf estimated expected loss} can be computed 
from the results of a learning experiment.

\item[\bf guess (for a test case) \hfill]
A prediction for the targets in a test case consisting of a single
value for each target, these values being chosen by the learning 
method with the aim of minimizing the expected {\bf absolute-error},
{\bf squared-error}, or {\bf 0-1 loss}.  If a {\bf no-guess penalty}
has been specified, a learning method also has the option of
making no guess for a particular target in a particular test case.

\item[\bf hierarchical testing scheme \hfill]
An experimental set-up in which separate, non-overlapping test sets are
used to assess the performance of a method as trained on different
training sets; distinguished from a {\bf common} testing scheme.

\item[\bf historical dataset \hfill]
A dataset that is included in the DELVE archive because it has been used
to assess learning methods in the past, but which is not recommended
for future use, except when there is a need to make comparisons with
past results in the literature.

\item[\bf input attribute \hfill]
For a particular {\bf prototask}, an attribute that is available
for use in predicting the values of the {\bf target attributes} in
the same case, but whose values do not themselves need to be predicted.

\item[\bf informative ordering \hfill]
An ordering of cases in a dataset (as originally obtained)
that conveys information that may be significant --- for instance,
an ordering of data on patients by date of admission to hospital.

\item[\bf integer attribute \hfill]
An attribute whose values are integers, and for which the
prior information does not specify an interpretation as
a {\bf categorical attribute}.  Note that the range of
an integer attribute may be restricted (eg, to the positive
integers).

\item[\bf learning experiment \hfill]
An experiment in which the performance of one or more 
{\bf learning methods} on one or more {\bf tasks} is assessed 
by applying the learning methods to several {\bf task instances}.
DELVE defines a standard scheme for conducting such experiments.

\item[\bf learning method \hfill]
A well-defined procedure for discovering relationships among
attributes on the basis of prior information and empirical data, and 
for making predictions for new cases using the relationships 
learned.  Learning can be {\bf supervised} or {\bf unsupervised}.

\item[\bf log-probability loss \hfill]
A loss function used with methods whose predictions are predictive
distributions over target values.  The log-probability loss is minus the 
log (base $e$) of the probability or probability density of the target
values.  This loss function can be used with any task, but for tasks
with real-valued targets (such as regression tasks), the loss must be 
computed by the learning method itself, rather than by DELVE.

\item[\bf loss function \hfill]
A measure of how far off a prediction is, given the actual values of
the targets. The standard loss functions DELVE supports are
{\bf squared-error loss}, {\bf absolute-error loss}, {\bf 0-1 loss}, 
{\bf squared-probability loss}, and {\bf log-probability loss}.  Specialized
loss functions can also be constructed that incorporate a 
{\bf no-guess penalty}, or that are based on a {\bf loss matrix}.

\item[\bf loss matrix \hfill]
For a prototask with one categorical target, a matrix that specifies 
the loss that is suffered for every possible combination of a guessed
value for the target and an actual value for the target.  For each
actual value of the target, the loss suffered when no guess is made
may also be specified.

\item[\bf missing value \hfill]
An indicator that the actual value of an attribute for a particular case
is not known.  In DELVE dataset files, a missing value starts with
a question mark; this may be followed by other characters to distinguish
values that are missing for different reasons.

%\item[\bf mysterious ordering \hfill]
%An ordering of cases in a dataset (as originally obtained) 
%that does not appear to be arbitrary, but whose significance
%(if any) cannot be determined from the available documentation.

\item[\bf natural dataset/prototask \hfill]
A dataset that comes from a real-world source, and for which there
is or was a real interest in learning relationships among the
attributes (for either scientific or engineering purposes).  A
prototask is classified as natural if it is based on a natural
dataset, and involves learning relationships that were of 
interest to the original investigators.

\item[\bf no-guess penalty \hfill]
The loss suffered when a learning method whose predictions take
the form of guesses decides to make no guess for a particular target
in a particular case.

\item[\bf nominal attribute \hfill]
A categorical attribute with at least three possible values (not 
counting missing values) for which the prior information does
not specify any natural ordering of the values.  An example
might be an attribute with values of ``beef'', ``pork'', and ``lamb''.

\item[\bf non-standard task instance \hfill]
A task instance in which the training and test sets are not
selected according the standard DELVE scheme.

\item[\bf noise level (for a target) \hfill]
The proportion of the variation in a target attribute that 
is not explained by the variation in the input attributes, even
given full knowledge of the true relationship between inputs and
targets.

\item[\bf order (of a dataset) \hfill] 
An indicator of whether the order of cases in the dataset (as originally
obtained) is {\bf informative} or {\bf uninformative}.

\item[\bf ordinal attribute \hfill]
A categorical attribute with at least three possible values (not
counting missing values) for which the prior information specifies a
natural ordering of the values.  An example might be an attribute with
values of ``no-education'', ``primary-education'', ``secondary-education'', 
and ``post-secondary-education''.

\item[\bf p-value (for a comparison) \hfill]
When comparing the estimated expected loss of two learning methods
on some task, the probability that a difference in estimated expected 
loss of equal or greater magnitude than that observed might arise by chance 
even if the true expected loss for the two methods is the same.  
A low p-value may give one confidence that the apparently better 
method actually is better.

\item[\bf passive value \hfill]
A value for a categorical attribute that is expected on the basis
of prior information to play a role different from that of the other
value or values of the attribute, with the passive value being
associated with a lack of positive influence.  If a
binary attribute has values of ``hockey-player'' and ``not-a-hockey-player'',
for example, ``not-a-hockey-player'' might be regarded as passive.

\item[\bf performance (of method) \hfill]
In the DELVE context, usually the predictive performance of the
method on some task, formalized in terms of {\bf expected loss}.
One might also be interested in the computational performance of a 
method (its time and memory requirements).

\item[\bf prediction (for a test case) \hfill]
The output of a learning method for a test case, embodying
the method's prediction regarding the likely values of the
targets in this case.  Predictions may be either single-valued 
{\bf guesses} for the target values, or {\bf predictive distributions} 
that say how likely each of the possible target values is.

\item[\bf predictive distribution \hfill]
A probability distribution produced by a learning method as its
prediction for the values of the targets in a test case.
For classification tasks, the predictive distribution consists of
a finite number of probabilities, which may be output in explicit form.
For tasks with real targets, the predictive distribution consists of
a probability density function, which DELVE does not attempt to
represent explicitly; instead, the learning method itself calculates 
the {\bf log-probability loss} based on its internal representation of 
the predictive distribution.

\item[\bf prior information \hfill]
Information regarding the the possible or likely nature of
the relationship being learned that is obtained from the
prior knowledge of the investigator (or a surrogate for the
investigator), rather than from the data itself.  

\item[\bf prototask \hfill]
A supervised learning problem associated with a {\bf dataset},
consisting of a set of {\bf target attributes} that are to be predicted, 
a set of {\bf input attributes} that may be used in making predictions, 
and a pool of {\bf cases} that are seen by the learning method.  A 
prototask can have many associated {\bf tasks}, in
which the available prior information  and the size of the training set are
also specified.  Prototasks are classified as {\bf natural}, {\bf cultivated}, 
{\bf simulated}, or {\bf artificial} according to their relationship to 
the real world.  {\bf Regression}
and {\bf classification} prototasks are distinguished by the nature of
their target attributes.

\item[\bf range (of attribute)\hfill]
The set of {\bf values} that an attribute could conceivably take on, including
the set of {\bf missing values} that are allowed for the attribute.

\item[\bf real attribute \hfill]
An attribute whose values are real numbers, and for which the prior 
information does not specify an interpretation as an {\bf angular},
{\bf integer}, or {\bf categorical attribute}.  Note that the range of 
a real attribute may be restricted (eg, to some interval).

\item[\bf relevance (of an input) \hfill]
The degree to which variation in an input attribute (within its
observed range) affects the values of the target attributes.
Put another way, the degree to which knowledge of the
input attribute's value helps in predicting the values of the
targets, given that the true nature of the relationship between
inputs and targets is known.

\item[\bf regression prototask/task \hfill]
A prototask (or task) in which all the targets attributes are
{\bf real}.

\item[\bf sequential prototask \hfill]
A prototask based on a dataset with an {\bf informative ordering}
in which this ordering has been preserved, and in which there may
therefore be {\bf dependencies} between nearby cases.

\item[\bf simulated dataset/prototask \hfill]
A dataset generated by a program (usually with a random component)
that simulates some actual phenomenon in a realistic fashion.
Prototasks based on such datasets are also referred to as simulated.

\item[\bf squared-error loss \hfill]
A loss function for regression tasks in which the loss is the
square of the difference between the guess and the
target.  When there is more than one target, the squared-error loss 
is the sum of such squared differences for all the targets.

\item[\bf squared-probability loss \hfill]
A loss function for classification tasks, used with methods 
whose predictions are predictive distributions over target values.
The squared-probability loss is the square of one minus the probability 
assigned to the correct target value, plus the sum of the squares of 
the probabilities assigned to all the other target values.  Squared-probability
loss cannot be used when there is more than one target attribute.

\item[\bf standard error (of estimate) \hfill]
The standard deviation of an estimate (eg, of expected loss) that
would be observed if the experiment on which the estimate is based
were to be repeated many times with new data randomly obtained
from the same source as the actual data.  (In practice, the
standard errors quoted are themselves estimates, since the 
true standard deviation usually depends on unknown quantities.)

\item[\bf standard task instance \hfill]
One of the task instances that are used in DELVE's standard scheme for learning 
experiments.

\item[\bf stratified training set \hfill]
A training set for a classification task in which training cases
have been selected in such a way that each of the different possible
target values appears the same number of times.

\item[\bf supervised learning \hfill]
Learning whose goal is to discover the relationship of certain
{\bf target attributes} to other {\bf input attributes}, and
on this basis predict the values of the target attributes for
a new case for which only the input attributes are known.

\item[\bf target attribute \hfill]
For a particular {\bf prototask}, an attribute whose values are to 
be predicted, based on the values of other {\bf input attributes} 
in the same case. 

\item[\bf task \hfill]
A specific learning context for a {\bf prototask}, consisting of the 
{\bf prior information} regarded as being 
available for use in learning, and the size and nature of the {\bf training set}
that will be provided.  A task is sufficiently
well specified that each learning method has a well-defined {\bf expected
loss} for a given task and loss function.  A task may be associated with
many {\bf task instances}, in which particular training sets and 
test cases are specified.

\item[\bf task instance \hfill]
A particular {\bf training set} for a {\bf task}, to which a learning method
can be applied as part of a learning experiment, together with a {\bf test
set} that is used to evaluate the accuracy of the learning method's
predictions. In DELVE's scheme for learning experiments, a set of {\bf standard
task instances} are defined; it is possible to define {\bf non-standard task
instances} as well.

\item[\bf test case \hfill]
A {\bf case} that is used to evaluate the performance of a 
learning method applied to a particular {\bf task instance}.

\item[\bf test set \hfill]
The set of all {\bf test cases} for a particular {\bf task instance}.
Note that although a task instance will normally include many test
cases, the predictions for the targets in each test case are to be made 
without using information from any other test case.

\item[\bf training case \hfill]
A {\bf case} that is part of the {\bf training set} made available
to a learning method.

\item[\bf training set \hfill]
The set of {\bf training cases} that are made available to a 
learning method in a particular {\bf task instance}.

\item[\bf uncontrolled attribute \hfill]
An attribute whose values were not fixed by the investigators who gathered
the data, but by some random process.  For example, the amount of rainfall
on various agricultural test plots would be an uncontrolled attribute 
(even though the investigators influence the amount of rainfall by where
they decide to put the plots).

\item[\bf uninformative ordering \hfill]
An ordering of cases in a dataset (as originally obtained) 
that does not convey any useful information --- for instance, 
a random ordering, or an ordering that is sorted by the value
of one of the attributes.

\item[\bf unsupervised learning \hfill]
Learning whose goal is to discover the relationships amongst
all attributes, without distinguishing some attributes as ``inputs''
and others as ``targets''.  DELVE does not currently
handle methods for unsupervised learning, but may do so in future.

\item[\bf value (of an attribute) \hfill]
The actual numerical or non-numerical quantity taken on by 
an {\bf attribute} in a particular {\bf case}.  Some cases may
have attributes with {\bf missing values}, for which the actual value is
not known, or with {\bf censored values}, for which the actual value is
known only to be beyond some given value.

\item[\bf 0-1 loss \hfill]
A loss function for classification tasks in which the loss is 0
when a guess matches the actual target value and 1 when the guess 
does not match the actual target value.  When there is more than one 
target, the total loss is the number of mis-matches between guesses 
and actual values.

\end{list}
